Method for generating a 3d printable model of a patient specific anatomy

ABSTRACT

A computer implemented method for generating a 3D printable model of a patient specific anatomic feature from 2D medical images is provided. A 3D image is automatically generated from a set of 2D medical images. A machine learning based image segmentation technique is used to segment the generated 3D image. A 3D printable model of the patient specific anatomic feature is created from the segmented 3D image.

BACKGROUND OF THE INVENTION

1 Field of the Invention

The field of the invention relates to a computer implemented method forgenerating a 3D printable model of a patient specific anatomy based on2D medical images.

A portion of the disclosure of this patent document contains material,which is subject to copyright protection. The copyright owner has noobjection to the facsimile reproduction by anyone of the patent documentor the patent disclosure, as it appears in the Patent and TrademarkOffice patent file or records, but otherwise reserves all copyrightrights whatsoever

2 Description of the Prior Art

Creating accurate 3D printed models of specific parts of a patient'sanatomy is helping to transform surgery procedures by providing insightsto clinicians for preoperative planning. Benefits include for examplebetter clinical outcomes for patients, reduced time and costs forsurgery and the ability for patients to better understand a plannedsurgery.

However, there is still a need to provide a secure platform which wouldenable the ordering and delivery of 3D printed models in a timely andcustomisable manner. Additionally, there is also a need to provide 3Dprintable models providing greater insight on the patient anatomy orpathology.

SUMMARY OF THE INVENTION

There is provided a computer implemented method for generating a 3Dprintable model of a patient specific anatomic feature from 2D medicalimages, in which a 3D image is automatically generated from a set of 2Dmedical images, a machine learning based image segmentation technique isused to segment the generated 3D image, and a 3D printable model of thepatient specific anatomic feature is created from the segmented 3Dimage.

Optional features in an implementation of the invention include any oneor more of the following:

-   -   The set of 2D medical images are images from the patient taken        from one or a combination of the following: CT, MRI, PET and/or        SPCET scanner.    -   2D medical images from multiple scanning techniques are        simultaneously processed    -   The set of 2D medical images is automatically pre-processed such        that important or critical features of the patient specific        anatomic features are made visible within the 3D printable model    -   Pre-processing of the 2D medical images is based on the specific        anatomic feature, specific pathology of the patient or any        downstream application such as pre-operative planning or        training purpose.    -   The set of 2D medical images is pre-processed to generate a new        set of 2D medical images which are evenly distributed according        to a predetermined orientation.    -   The predetermined orientation is based on the patient specific        anatomic feature, specific pathology of the patient or any        downstream application such as pre-operative planning or        training purpose.    -   The predetermined orientation and spacing between each 2D        medical image within the new set of 2D medical images are        determined using machine learning techniques.    -   The predetermined orientation and spacing between each 2D        medical image within the new set of 2D medical images are user        configurable.    -   A missing slice from the set of 2D medical images is        automatically detected    -   A 2D image corresponding to the missing slice is generated using        interpolation techniques.    -   The segmentation technique is based on one or a combination of        the following techniques: threshold-based, decision tree,        chained decision forest and a neural network method.    -   Voxel based classification technique is used in which voxel        information from each axis or plane is taken into account.    -   The likelihood of a voxel of the 3D image having properties        similar to the patient specific anatomic feature is calculated        using a logistic or probabilistic function.    -   A neural network determines a weight for each axis or plane in        each voxel of the 3D image    -   The segmentation technique is further improved using        multi-channel training    -   Each channel represents a 2D image corresponding to a slice        position within the 3D space of the 3D image.    -   A channel is represented using a ground truth image.    -   A 3D mesh model of the patient specific anatomic feature is        generated from the segmented 3D image, and the 3D printable        model is generated from the 3D mesh model    -   The 3D mesh model is further processed using finite element        analysis.    -   Points or areas in the 3D mesh model requiring further post        processing steps are automatically detected.    -   Further post processing steps include placement of a dowel or        other joining structure.    -   The optimal placement of a dowel or other joining structure is        automatically determined.    -   Parameters of the patient anatomic feature are automatically        determined from the analysis of the generated 3D image, such as        volume or dimensions of the anatomic feature, thicknesses of the        different layers of the anatomic feature.    -   Specific anatomic feature is a heart and the measured parameters        include one of the following: volume of the heart, volume of        blood in each chamber of the heart, thickness of the different        layers of the heart wall, size of a specific vessel.    -   The 3D printable model is 3D printed as a 3D physical model such        that it represents a scale model of the patient specific        anatomic feature such as a 1:1 scale model or a more appropriate        scale model such as a reduced scale or enlarged scale model of        the patient specific anatomic feature depending on the intended        downstream application.    -   The 3D printable model is 3D printed with critical or important        features of the specific anatomic feature made easily visible or        accessible.    -   A 3D mesh is generated from the set of 2D medical images, in        which the 3D mesh is a polygonal representation of the volume of        the patient specific anatomic feature.    -   A line or spline is extracted from the 3D mesh along a direction        of the patient specific anatomic feature.    -   A classifier is used to identify the anatomic feature from the        extracted line or spline.    -   The method further includes the step of generating a wireframe        model of the 3D mesh.    -   A classifier is used to identify the physical properties of the        anatomic feature from the extracted line or spline.    -   A classifier is used to identify a pathology of the anatomic        feature from the extracted line or spline.    -   The classifier is trained to identify a specific anatomical        feature.    -   The classifier is trained to determine parameters of the        specific anatomic feature such as its location relative to the        human body, dimension or thickness.    -   The classifier is trained to determine a potential defect or        pathology of the specific anatomic feature.    -   The classifier is a principle component analysis classifier.    -   The method further includes the step of splitting the 3D        printable model into a set of 3D printable models, in which the        set of 3D printable models include connective pieces, in which        the location of each connective piece is automatically        generated.    -   The 3D printable model is decided based on the patient specific        anatomy and pathology.    -   Splitting of the 3D printable model cannot be decided only on        assessing the surface of the patient specific anatomy.    -   A connective piece is a magnetic or metal element.    -   Each connective piece is located such that a set of 3D printed        physical models from the set of 3D printable models can be        connected to represent the patient specific anatomic feature and        is prevented from being wrongfully connected    -   The set of 3D printed physical models represent a scale model of        the patient specific anatomic feature such as a 1:1 scale model        or a more appropriate scale model such as a reduced scale or        enlarged scale model of the patient specific anatomic feature        depending on the intended downstream application.    -   Critical or important features of the specific anatomic feature        are made easily visible within the set of 3D printable physical        models.    -   Critical or important features of the specific anatomic feature        are made easily accessible within the set of 3D printable        physical models.

Another aspect is a 3D physical model representing a scale model of apatient specific anatomic feature that is 3D printed from the 3Dprintable model generated from the method steps defined above, in whichthe scale model is a 1:1 scale model or a more appropriate scale modelsuch as a reduced scale or enlarged scale model of the patient specificanatomic feature depending on the intended downstream application

Another aspect is a computer implemented system for generating a 3Dprintable model of a patient specific anatomic feature from a set of 2Dmedical images, the system comprising a processor for automaticallygenerating a 3D image from the set of 2D medical images, segmenting thegenerated 3D image using a machine learning based image segmentationtechnique, and outputting a 3D printable model of the patient specificanatomic feature from the segmented 3D image.

Another aspect is a computer implemented method for printing a 3D modelof a patient specific anatomic feature comprising: uploading a set of 2Dmedical images to a server, processing at the server the set of 2Dmedical images into a 3D printable model of the patient specificanatomic feature; the server transmitting instructions for printing the3D printable model to a printer, in which a security engine validatesthat the 3D printable model is associated with the correct patient data,and in which an end-user located at a remote location from the printermanages the printing of the 3D printable model.

BRIEF DESCRIPTION OF THE FIGURES

Aspects of the invention will now be described, by way of example(s),with reference to the following Figures, which each show features of theinvention:

FIG. 1 shows a diagram illustrating the Axial3D system workflow.

FIG. 2 shows a diagram illustrating hashing of the file of the 3Dprintable model.

FIG. 3 shows a set of DICOM stack images with pixels indicated as boxes.

FIG. 4 shows a set of DICOM stack images and a 3D image with voxelsindicated.

FIG. 5 shows a 3D image of making selections in the voxel space.

FIG. 6 shows a specific voxel from 3 orthogonal planes.

FIG. 7 shows data registration of two different datasets for a singlepatient.

FIG. 8 shows a diagram illustrating equidistant slices in a particularplane.

FIG. 9 shows a diagram illustrating the multi-channel training.

FIG. 10 shows a diagram illustrating the multi-channel training.

FIG. 11 shows a wireframe model of the mesh for a specific anatomy.

FIG. 12 shows diagrams of the wireframe model, the anatomy, and of anoverlaid model of the anatomy with a verified wireframe model.

FIG. 13 shows a 3D bone model with a spline.

FIG. 14 shows three splines of a bone.

FIG. 15 shows a 3D printable model of a heart.

FIG. 16 shows a 3D physical model of a heart printed in two parts.

DETAILED DESCRIPTION

This Detailed Description section describes one implementation of theinvention, called the Axial3D system.

FIG. 1 shows a diagram illustrating the Axial3D system workflow ofordering a 3D printed Model. The Axial3D system uses machinelearning-based techniques to automatically produce patient-specific 3Danatomical models based on a patient's scans.

The 3D anatomical models may be generated, printed and delivered in24-48 hours.

As shown in FIG. 1, a 3D print is requested via Axial3D dedicated portalas follows:

-   -   Register on theAxial3D ordering platform        https://orders.axial3d.com/ (10);    -   Select New print and complete patient details such as birth        date, gender, anatomical region of interest, dispatch date,        anatomical model service required, material type, pathology        description, lead consultant details;    -   Send request to PACS manager or radiologist or upload DICOMS        themselves (11);    -   3D annotation or written description is given of request Data is        proceed by Axial3D software or personnel into a 3D printable        file (12);    -   SLT/OBJ of final print ready file is sent securely via a VPN to        a 3D printer on site with each printer having its own wireless        (13);    -   3G/4G network that can send a receive data to Axial3D's web        application;    -   If a print order is too large for internal capacity—Axial3D        prints and sends a 3D model to customer.    -   If orders are too large or complex Axial3D prints are managed by        Axial3D's printing service (14).

As an example, a clinician or radiologist may order a 3D print of apatient specific anatomic feature via the web portal. The Axial3D systemthen automates the entire steps of the 3D printing process fromprocessing 2D medical images to sending instructions to a 3D printer forprinting the patient specific anatomic feature. The clinician is thenable to receive the 3D physical model in a timely manner, such as in 24hours or 48 hours from placing the order, with minimum or zeroinvolvement from his part. The Axial3D system also provides theclinician with an additional report of the specific anatomic featurealongside the 3D physical model based on a detailed analysis of thespecific anatomic feature.

Cybersecurity Process in Medical 3D Printing

We have developed a digital platform to enable the secure and verifiableproduction and delivery of 3D printed anatomical models on demand and todeliver this globally, at scale and in a wide range of scenarios: makingit available not just to health authorities, private hospitals andsurgeries but ultimately any hospital. The technological challenge is toprovide indisputable verification of the provenance of both the virtualmodel generated from a patient's anonymised data and any physical modelthat is 3D printed from it. The stakeholders involved in this processrepresent multiple parties spread across multiple organisationstherefore they need to be reliably identified, authenticated and capableof independently verifying the provenance of these models.

This enables remote printing of 3D anatomical models, where the printingis done in one location and controlled remotely in another location.Once 3D physical models are ordered, 3D models are generated from 2Dmedical scans, and are then remotely reviewed, approved and controlledby a 3D printing technician.

The 3D printing technician may also control more than one printerremotely and the system is automatically able to decide how best toselect or arrange the printing on the one or more printers.

The cybersecurity process is crucial in order to prove or validate thatthe printed 3D physical object is the one that was sent remotely andthat it is associated with the correct patient without disclosing anypatient confidential data.

Crypto Signing of Files

We create and store a hash of the 3D model file representing the 3Dprintable model of a specific anatomic feature and use that to recreatethe object or 3D physical model anytime that it is required. This hashcan be used to quickly establish if the file has been modified.

Every time we upload or make changes to the file on the web app we needto create a new hash however the one that is created at the end of theprocess is a canonical hash for the printed file. Therefore all previousfiles are quality controlled ‘drafts’. The canonical is the file that wepublish so that the user has the end file.

In the process of generating an anatomical model from medical scans thedata undergoes a number of transformations and modifications. A hashfile is generated at each of these steps in order to record thesechanges. The process of identifying anatomy in the scan produces labelson the scan that are subsequently used to generate a print file. Thehashing process records this and acts as a history of the changes.Modifications to the file are stored and used to provide a trace of theprovenance of the file. In this way the user can be assured of theprovidence of the file that they are using.

We have implemented a system that allows for the crytographic signing offiles and their subsequent distribution. The distribution of files forprinting is managed by providing a decentralised file signing service.This is done by cryptographically signing the files using private/publickey based encryption. This allows the verification of files by remoteparties in a secure manner.

A service is provided that allows the download of the file and of anysubsequent testing of the files for correctness. Files can be stored onobject file system like S3 along with hash of file. A ‘central’repository of hashes then links the file to the order. This repositorymay be a file, a database or a distributed ledger.

FIG. 2 shows a diagram illustrating the tracking of modifications of thefile, in which changes to a file are committed to the repository, andchanges to an instance of the repository are synchronised betweenrepositories.

Our system ensures that only validated files can be printed. Files aresigned and only those that have passed the cryptographic challenge areaccepted for printing. As a result only files that have been signed andverified against the verification server can be sent to the printer.This also means that all files can be encrypted both at rest and attransfer and that modifications can be recorded and observed withoutneeding to see the contents of the file.

Our system may sit in front of printers ensuring that only encryptedfiles are sent for printing. Files can be decrypted in transit as theprint is being completed and ensuring that only encrypted versions ofthe file are ever stored/transmitted.

Working Natively in 3D Space

Most segmentation methods work on applying algorithms to 2D images and3D models are then generated from the segmented 2D images.

FIG. 3 shows a set of DICOM stack images with pixels indicated as boxes.FIG. 4 shows a set of DICOM stack images (40) and a 3D image with voxelsindicated (41). We work natively in the 3D space by converting the scanslices shown in FIG. 3 into a single volume shown in FIG. 4. We computethe zero point of our coordinate system for the volume and orientate allslices in the scan to this. This allows us to calculate the alignment ofslices with reference to the volume and observe properties of the set ofslices. This means that we now natively work in 3D using 3D featuredetection filters, essentially becoming a voxel classification ratherthan pixel based classifier

In FIG. 5 a knee is shown and specific voxels are highlighted. One keyadvantage of working natively in 3D is that the system incorporatesorthogonal information in the scoring metric. This is most simplyindicated by considering FIG. 6 where 3 planes are considered together.A particular voxel is shown with three planar cuts through that voxelwhich reveals more information about the likelihood of a voxel being amember of a specific class or not. By incorporating information from allplanes for each voxel it is possible to identify junctions between bonesor other sections of anatomy more effectively. This is becausetransitions in the image (e.g. voxel intensity) are easier to spot whenconsidering all planes. The result is that all voxels in the scan can beused simultaneously to train the algorithm. In practice this means thatlarger, spatial and biological features can be encoded in the algorithmto overcome specific challenges at anatomical intersections such asmyocardial wall to ventricle (heart) or bone joints such as those shownin FIG. 5

Combining Image Registration and Making Multi Modal Inference

We can register multiple image stacks and modalities (such as MIRi & CTor Mri and Mri where different structures are highlighted in moredetail) scans to overlay the voxels of the different scans as shown inFIG. 7 in which data registration of two different datasets for a singlepatient is illustrated. We can identify landmarks within the scans tofacilitate mapping pixels from one scan to another. A landmark is apoint or shape that is shared between individuals by common descent. Itcan be biologically meaningful such as the shape of the eye corner ofthe skull or mathematically expressed as the highest curvature point ona bone's surface. This means that information from the multiple scanscan be used simultaneously to identify features for the machine learningalgorithm Since both modalities can be thought of as different views ofthe same anatomy the combination allows us to add additional informationinto the training phase. In this way 2D medical images, provided forexample from CT, MRI, or PET scans, can be processed together.

The Axial3D system includes the steps of (i) receiving 2D medicalimages, (ii) automatically generating a 3D image from the 2D medicalimages, and (iii) processing the 3D image in order to segment orclassify the 3D image. A 3D printable model can then be generated fromthe segmented 3D image.

The 3D image data file format includes for example any point cloudformat or any other 3D imaging format.

Key features of the system are, but not limited to, the following:

-   -   Anatomical transitions are easier to identify since the 3D image        includes image data from more than one direction. By comparison,        information from only one direction is available when working        with slices of 2D images.    -   Consequently, this also improves the identification of a        specific anatomy.

Combining Information from Multiple Planes

In order to image a specific anatomy, cross-sectional images are takenat any angle. As an example, an MRI scan of the heart takes 2D images atdifferent directions Working natively in 3D improves the accuracy (asmeasured using standard metrics such as but not limited to the DICEcoefficient) of the generated 3D printable model. On a per voxel basisthe accuracy of the prediction is improved by considering the 3Dproperties of the voxel over considering the 2D properties of the pixelsand combining them. Each plane or 2D image and it's constituent pixelsbecome features of the 3D volume. For each voxel the features of eachpixel are encoded as features of the voxel. The network is then trainedand determines the appropriate weight to be given to each plane. Eachfeature is represented as a discrete range and is optimised by theneural network training process. In this way it is possible for thetraining process to learn the appropriate integration of the additionalinformation from all planes.

Post Segmentation Utility of Anatomical Feature Delineation

When a piece of anatomy has been fully and accurately segmented it ispossible to carry out measurement of a number of physical properties ofthe anatomy, for example the heart. The segmented anatomy can bemeasured by relating the pixel size to a physical scale from thecoordinate system.

Parameters of the anatomic features are determined, such as, but notlimited to:

-   -   The volume of the anatomical region;    -   The volume in a cavity of the anatomical feature e.g. the blood        in each chamber of the heart;    -   The thickness of the different layers of the anatomical feature        e.g. the heart wall,    -   The size and diameter of a feature, e.g. a blood vessel or bone;    -   The directional properties of a shape e.g. in scoliosis        cases—detection of scoliosis and type of scoliosis, measuring or        determining the angle or degree of curvature;    -   Cortical bone density—determination whether a screw is able to        fit, or automatically determine the parameters of a screw that        fits,    -   Aneurysm—will a coil or a clip fit be able to stop or block        blood flow,    -   Force needed to break a bone,    -   Further information on the extent of a pathology or injury    -   Information on an additional pathology that was not reported by        the clinician, such as the location of a previously unknown        fracture.

When a 3D printable model is ordered, the system produces and sends areport to the physician with the above information. This can improve asurgeon's preoperative planning, and further reduce costs to anhealthcare provider. For example, from understanding vessel size moreaccurately, a surgeon may then make an informed choice for the rightstent size prior to surgery. The system may also automaticallydetermines the parameters of the stent.

Automatically Identify and Repair Spacial Errors and Inconsistencies inthe Volumetric Data

FIG. 8 shows a diagram illustrating equidistant slices in a particularplane We are applying two methods one for identification of nonequilinear slices and one for missing slices. The trajectory of the 2Dslices is plotted and analysed. If a slice is found above or below acertain trajectory threshold, then it is removed from the analysis priorto the generation of the 3D image. The slices must be in planes that arecongruent with respect to each other, they are occurring parallel and atan equal distance apart with respect to the base plane. Assuming thatthere exists a set of equilinear slices and we can identify such a setand such a set minus of the slices. We are therefore fault tolerant inthe identification of equilinear slices to one.

Combining Interpolated Data from Multiple Slices Containing Slices fromMultiple Angles.

We then have developed a method for inferring the missing data betweentwo slices. This relies on the ability to create a missing slice withthe correct 3D geometry and interpolated pixel values.

Many medical imaging datasets contain slices of the patient frommultiple angles. While CT scanning is typically limited in its abilityto obtain slices at standard angles, oblique scans are routinelyacquired for MR scans. Oblique scans are often used in MR imaging inorder to minimise the number of total images to be collected andtherefore reduce the time and cost of performing a scan. Typically, whensuch technique is used, a relatively small number of slices is acquiredat each oblique angle (typically 5 to 10 images) at large slice spacing(5 to 10 mm); the oblique scans are often taken at either three nearlyperpendicular directions (axial, coronal, sagittal) plus an additionaloblique axis, however, imaging angles and number of scans are to thediscretion of the medical professional.

As a consequence, too few slices along a single axis may be provided togenerate a complete volume of high enough quality. For example, thespacing between each slice may be greater than five millimetres,entirely missing important anatomical features.

Resulting images may only provide sufficient visual information on aspecific lesion when viewed in combination: each portion of the lesionmay be located in the large gaps of one of the scans, while it may bevisible in the other ones. For example, a 10 mm tumor mass may bevisible only in one slice of the axial scan, one of the coronal scans,and two slices of the sagittal scan; in this scenario, the oncologistwill view the four images at the same time to obtain a 3 dimensionalunderstanding of the tumor shape and volume.

The Axial3D system is able to automatically make decisions on how toprocess the 2D medical images in order to provide an accurate 3Dphysical print of a specific anatomic feature in which any critical orimportant features of the specific anatomic feature are made visible.These critical or important features may also be made readily accessibleby splitting the 3D physical model into a set of connectable 3D physicalmodels. These processing decisions may be based on the specific anatomicfeature, a specific pathology or any other pre-configured or learntparameter. This, in turn, aids in the diagnosis and treatment ofpatients, improving surgical planning and patient care.

In this method we show how to interpolate multiple simultaneous stacksinto one volume. This leverages the intersecting slices to achievehigher information density and create a highly fidelity interpolation.The slice spacing for the reconstructed volume is limited by theoriginal oblique scan spacing: depending on the number of oblique scans(typically 3 or four as mentioned above), the slice spacing of thereconstructed volume can be as low as a fifth of the original scan (egif the oblique scans slice spacing varies between 5 and 6 mm, thereconstructed volume spacing can be as low as 1 mm).

The interpolation was achieved by finding the absolute positions of thecorners of each DICOM image in each stack relative to origin determinedby the scanner itself and reported in the DICOM header. This allowed abounding box to be constructed to encompass all of the images in a spacein which they are all embedded. By discretizing the bounding box so thatit represented volume of voxels spanning the dimensions of all of thestacks, a mapping could be determined from the space of each stack ofDICOMs to the new volume space. At each point in the new volume, theclosest pixels K in the DiCOMs to that point were determined and theirdistances d computed. The voxel value M at this point was then computedas the weighted sum:

$M_{\text{?}{jk}} = {{\frac{\Sigma_{\text{?}}K_{\text{?}}q_{\beta}}{\Sigma_{\text{?}}q_{\beta}}{where}\mspace{14mu} q} = d^{- 1}}$?indicates text missing or illegible when filed

For each imaging orientation a stack of images was given as part of theoriginal dataset and for each orientation there were 20-30 such stacksrepresenting scans taken at those same locations but at different times.Each interpolation was generated for a series of DICOM images across allorientations of scan but for one time stamp.

This makes for a three dimensional interpolation. Hence, the original 2Dslices from multiple angles are transformed into a set of evenlydistributed parallel 2D slices prior to the generation of the 3D image.

Multi-Channel Training

Here we describe the addition of “above and below” slices alongside atypical input image to improve the segmentation network. This informsthe neural network about continuous structures and those that are justspurious artefacts of a particular scan We anticipate improvements inthe neural network specifically at correctly identifying thinner bonefilaments while simultaneously removing areas of an image that havesimilar Hounsfield values but aren't the same category of anatomy. Forthe three-channel example, the neural network would need to take inputsof the shape: (batch_size, channels, X, Y)

The data is split in order to fit into the required memory size. Thesplit data may then be fed into any neural network, or any imageanalysis platform.

To achieve this, each stack was first padded with an ‘image of zeros’ onthe top and bottom of the stack. This meant that groups of three slicescould be formed into an object with the same total number of inputobjects, as shown in FIG. 9.

Each input triplet will have a ground truth or gold standardcorresponding to the ground truth image associated with the centralimage, in order to give the “above and below” information, as shown inFIG. 10. Each image and ground truth pair will still exist when theextra channels have been added. The same principle applies for anynumber of odd channels; for every two more channels, another layer ofpadding should be added to retain the same number of inputs. Thesituation is slightly trickier when dealing with an even number ofchannels, but this is less desirable because it removes the nice aspectof symmetry. In practise, it might also be more useful to add a paddingthat corresponds to the minimum Hounsfield value of the stack, becausethis avoids a very strong transition which might hinder learning. In thecase where an image has a padding image above or below it, there issimply less useful information to make a prediction with, the presenceof the padding should not affect the prediction itself.

Examples of extracted 3D features are the following:

-   -   transition;    -   pixel intensity;    -   shapes;    -   3D shapes;

‘Wireframe’ Shape Detection

We generate the isosurface of the anatomical feature by transforming theprobability distribution matrix from the inference algorithm into adiscrete scalar volume. This is then used to generate a volumetric meshthat is a polygonal representation of the volume of the anatomicalfeature. After the surface is generated we draw a wireframerepresentation of the surface. This is composed a series of splines thatform an outline of a given surface mesh. These can be compared toexisting mesh outlines to see if they match.

FIG. 11 shows a wireframe model of the mesh for a specific anatomy.

FIG. 12 shows diagrams of the wireframe model, the anatomy, and of anoverlaid model of the anatomy with a verified wireframe model.

Building a wireframe model of the mesh helps to quickly identify aspecific shape and its location in relation to the body. This, in turnimproves the accuracy of the 3D printable model and of the 3D printedphysical model.

Checking a line in one dimension to compare shapes is lesscomputationally intensive than checking a 3D surface to compare shapes.In addition, checking for a continuous line helps in identifyingcontinuous anatomy, whereas checking for a 3D surface is more prone toerrors.

Simple method for determination of hone class Lines can be drawn alongthe surface of anatomy that provide a unique identifier of the landmarkson the surface of the anatomy. ML models can be trained to identify setsof peaks and troughs in the surface line and relationships between themthat allow for the classification of these surface lines and thereforethe identification of anatomy.

Wireframe representation of the mesh. It is possible to draw the singlelines that form splines along the length of each bone in the scene, asshown in FIG. 13 where a line (131) from the wireframe model shows aspline of a bone.

FIG. 14 shows a spline of a first bone, a spline of a second bone and aspline of a third bone.

The splines above show two different bones—spline 2 and 3 are the samebone in different people. A classifier can be trained to identifybetween the two splines. The classifier can include a PCA (PrincipleComponent Analysis) classifier.

Orientation Fixing

-   -   Detecting overlap in shapes/volumes/meshes.    -   Allowing comparison between these shapes in 3D space.    -   Registration of images to the origin from DICOMs from two        different scans (e.g. one MRI and one CT). This can be achieved        in a number of ways; if the both sets of images make reference        to the same origin point then it is possible to simply overlay        the scans. However, if these are not present the algorithm will        detect the anatomy in both scans and uses 3D object retrieval        techniques to overlay the anatomy and recognise the same parts        in the two scans. These can be combined with conventional        technique from 2D registration to provide a higher level of        confidence.

Auto Detect where to Place Dowels and Other Post Processing Steps

We carry out shape modeling whereby we determine the weakest andstrongest position on the mesh. This can be achieved by bending anddistorting the mesh and determining the points of maximum and minimumflex. The output of this stage will be a heatmap of the mesh, whichprovides a score of the strength of the mesh at a given point. Thisallows us to identify areas that require strengthening. It will alsoallow us to detect places that can be used for the placement of magneticconnections.

We have developed an algorithm that allows us to determine points ofarticulation in a 3D mesh. This is used by us to determine where themodel should have additional support structures applied. We applyuniform vertical pressure on the mesh and identify the degree ofrotation of the polygons upon application of pressure. Points orpolygons that rotate by 90 degrees or more are in the most need offurther reinforcement. Finite element analysis can be applied to the 3Dmesh to develop a map of the mesh that captures structural properties ofthe mesh. This information can then be used to detect positions on themesh that can be used to deploy dowels and other joining structures.

We have implemented heuristic algorithms that allow us to effectivelyenumerate the potential solutions to the problem and identify best fitsolutions We have defined criteria for the placement of dowels assupport structures between parts of our models. We then use these asrules for optimisation of placement of such support structures. Weemploy wave functions to identify and optimise the placement of dowelsand other structures in the 3D mesh. These are then solved by wavefunction collapsing which produces the optimal location of the dowel.Additional constraints can be placed on the solution that avoidparticular features identified by the user.

Another use case is where we have split the model in two or more piecesand desire to reattach using magnets. We have developed an algorithmthat allows us to identify the optimal location of these attachingmagnets. This is an extension of the above algorithm whereby we add afurther constraint on the torsion, squishing or twisting of the modelthat captures the property of the magnet.

Deconstructed Anatomy with Magnetic Connections

User defines split line through whole model or splits model through anon-uniform cut to separate specific pieces of anatomy (e.g. pubis andilium from ischium within the hemi pelvis). The user then inputsdiameter and depth of magnets and software automatically embeds magnetindents into surface of anatomy or if walls are too thin incorporatescylindrical inset on the exterior of model (embedded and cylindricalinset models below).

Parts are split such that it is not possible to connect the differentparts together the wrong way. Magnetic or metal elements are placed toguide the different parts together. The metal elements are magneticallyattracted to the element located to another part such that it is notpossible to connect the different parts incorrectly.

As an example, FIG. 15 shows a 3D printable model of the heart. FIG. 16shows a 3D physical model of the heart printed in two separate parts.This enables a physician to view the 3D printed physical anatomy as awhole while at the same time being able to open it and see what isinside. The printed model can then be put together again knowing it hasbeen put together in the correct way.

The different parts may be printed in different colors or with differentmaterial formulations i.e. soft and hard polymers.

Key Features

This section summarises the most important high-level features, animplementation of the invention may include one or more of thesehigh-level features, or any combination of any of these. Note that eachfeature is therefore potentially a stand-alone invention and may becombined with any one or more other feature or features.

We Organise these Features into the Following Categories:

A. Working natively in 3D

B. Wireframe model

C. Splitting a 3D printable model into a set of 3D printable models

D. Remote printing

A. Working Natively in 3D

A computer implemented method for generating a 3D printable model of apatient specific anatomic feature from 2D medical images, in which:

-   (a) a 3D image is automatically generated from a set of 2D medical    images;-   (b) a machine learning based image segmentation technique is used to    segment the generated 3D image; and-   (c) a 3D printable model of the patient specific anatomic feature is    created from the segmented 3D image.

Optional:

-   -   The set of 2D medical images are images from the patient taken        from one or a combination of the following: CT, MRI, PET and/or        SPCET scanner.    -   2D medical images from multiple scanning techniques are        simultaneously processed.    -   The set of 2D medical images are automatically pre-processed        such that important or critical features of the specific        anatomic feature are made visible within the 3D printable model.    -   Pre-processing of the 2D medical images is based on the specific        anatomic feature, specific pathology of the patient or any        downstream application such as pre-operative planning or        training purpose.    -   Pre-processing of the 2D medical images is based on the specific        anatomic feature or specific pathology of the patient.    -   The set of 2D medical images is pre-processed to generate a new        set of 2D medical images which are evenly distributed according        to a predetermined orientation.    -   The predetermined orientation is based on the patient specific        anatomic feature, specific pathology of the patient or any        downstream application such as pre-operative planning or        training purpose.    -   The predetermined orientation and spacing between each 2D        medical image within the new set of 2D medical images are        determined using machine learning techniques.    -   The predetermined orientation and spacing between each 2D        medical image within the new set of 2D medical images are user        configurable    -   In which a missing slice from the set of 2D medical images is        automatically detected.    -   A missing slice is corrected by generating an image        corresponding to the missing slice using interpolation        techniques.    -   The segmentation technique is based on one or a combination of        the following techniques: threshold-based, decision tree,        chained decision forest or a neural network method;    -   Voxel based classification technique is used in which voxel        information from each axis or plane is taken into account.    -   The likelihood of a voxel of the 3D image having properties        similar to the patient specific anatomic feature is calculated        using a logistic or probabilistic function.    -   The neural network determines a weight for each axis or plane in        a voxel of the 3D image.    -   Segmentation technique is further improved using multi-channel        training.    -   In which each channel represents a 2D image corresponding to a        slice position within the 3D space of the 3D image.    -   3D mesh model of the patient specific anatomic feature is        generated from the segmented 3D image.    -   3D mesh model is further processed using finite element        analysis.    -   points or areas in the 3D mesh model requiring further post        processing steps are automatically detected.    -   Further post processing steps include placement of a dowel or        other joining structure.    -   In which the optimal placement of a dowel or other joining        structure is automatically determined.    -   3D printable model is based on the generated 3D mesh model.    -   The 3D printable model is 3D printed as a 3D physical model such        that it represents a scale model of the patient specific        anatomic feature such as a 1:1 scale model or a more appropriate        scale model such as a reduced scale or enlarged scale model of        the patient specific anatomic feature depending on the intended        downstream application.    -   The 3D printable model is 3D printed with critical or important        features of the specific anatomic feature made easily visible or        accessible.    -   Parameters of the patient anatomic feature are determined from        the generated 3D image, such as volume or dimensions of the        anatomic feature, thicknesses of the different layers of the        anatomic feature.    -   The specific anatomic feature is a heart and the measured        parameters are: volume of the heart, volume of blood in each        chamber of the heart, thickness of the different layers of the        heart wall, size of a specific vessel.

B. Wireframe Model

Computer implemented method for identifying an anatomic feature from aset of 21) medical images, the method includes:

-   -   (a) generating a 3D mesh from the set of 2D medical images, in        which the 3D mesh is a polygonal representation of the volume of        the anatomic feature;    -   (b) extracting a line or spline from the 3D mesh along a        direction of the anatomic feature; and    -   (c) using a classifier to identify the anatomic feature from the        extracted line or spline.

Optional:

-   -   The classifier is used to identify the physical properties of        the anatomic feature from the extracted line or spline.    -   The classifier is used to identify a pathology of the anatomic        feature from the extracted line or spline.    -   The method further includes the step of generating a wireframe        model of the 3D mesh.    -   A 3D image is automatically generated from the set of 2D medical        images and the 3D mesh is generated from the segmentation of the        3D image.    -   The classifier is trained to identify a specific anatomical        feature.    -   The classifier is trained to determine parameters of the        specific anatomic feature such as its location relative to the        human body, dimension, thickness.    -   The classifier is trained to determine a potential defect or        pathology of the specific anatomic feature.    -   The classifier is a principle component analysis classifier.

C. Splitting a 3D Printable Model into a Set of 3D Printable Models

Computer implemented method of splitting a 3D printable model of apatient specific anatomic feature into a set of 3D printable models, inwhich the method comprises the step of automatically splitting the 3Dprintable model into a set of 3D printable models, in which the 3Dprintable models include connective pieces, where the location of eachconnective piece has been automatically generated.

Optional:

-   -   Splitting of the 3D printable model is decided based on the        patient's pathology and anatomy. Whereby, information cannot be        gained from assessing the surface of the given structure alone.    -   Connective piece is a magnetic or metal element;    -   Each connective piece is located such that a set of 3D printed        physical models from the set of 3D printable models can be        connected to represent the patient specific anatomic feature and        is prevented from being wrongfully connected.    -   The set of 3D printed physical models represent a scale model of        the patient specific anatomic feature such as a 1:1 scale model        or a more appropriate scale model such as a reduced scale or        enlarged scale model of the patient specific anatomic feature        depending on the intended downstream application    -   critical or important features of the specific anatomic feature        are made easily visible within the set of 3D printable physical        models.    -   critical or important features of the specific anatomic feature        are made easily accessible within the set of 3D printable        physical models.

D. Remote Printing

A computer implemented method for printing a 3D model of a patientspecific anatomic feature comprising:

(a) uploading 2D medical images to a server,(b) processing at the server the 2D medical images into a 3D printablemodel of the patient specific anatomic feature; and(c) the server transmitting instructions for printing the 3D printablemodel to a printer, in which a security engine validates that the 3Dprintable model is associated with the correct patient data;in which an end-user located at a remote location from the printermanages the printing of the 3D printable model.

Optional:

-   -   2D medical images and additional metadata are anonymised prior        to being sent to the server such that no identifiable healthcare        or personal information is transferred to the server.    -   The end-user remotely schedules, initiates or approves the        printing of a 3D printable model on one or more printers via a        Web application.    -   The end user remotely controls one or more printers and the        printing is automatically arranged on the one or more printers.    -   A hash of a file corresponding to the file of the 3D printable        model is created and stored within a central repository.    -   The central repository is accessed by the server, in which the        central repository is a file, a database or a distributed        ledger.    -   The hash is used to recreate or validate the printing or any        subsequent printing of the 3D patient specific anatomic feature.    -   Modifications to the file are stored with the hash and used to        provide an audit trail of the provenance of the file.    -   The hash is used to establish if a file has been modified.    -   the distribution of one or more files for 3D printing one or        more specific anatomic features is managed by a centralised file        signing service.    -   Files corresponding to the 3D printable model are encrypted        using private/public key based encryption.    -   The security engine ensures only encypted files are transmitted        for printing.    -   Files are only decrypted in transit as a print is being        completed.

Note

It is to be understood that the above-referenced arrangements are onlyillustrative of the application for the principles of the presentinvention. Numerous modifications and alternative arrangements can bedevised without departing from the spirit and scope of the presentinvention. While the present invention has been shown in the drawingsand fully described above with particularity and detail in connectionwith what is presently deemed to be the most practical and preferredexample(s) of the invention, it will be apparent to those of ordinaryskill in the art that numerous modifications can be made withoutdeparting from the principles and concepts of the invention as set forthherein.

1. A computer implemented method for generating a 3D printable model ofa patient specific anatomic feature from 2D medical images, the methodcomprising: automatically generating a 3D image from a set of 2D medicalimages; using a machine learning based image segmentation technique tosegment the generated 3D image; and creating a 3D printable model of thepatient specific anatomic feature from the segmented 3D image.
 2. Themethod of claim 1, wherein the set of 2D medical images are images fromthe patient taken from one or a combination of the following: CT, MRI,PET and/or SPCET scanner.
 3. The method of claim 1, wherein 2D medicalimages from multiple scanning techniques are simultaneously processed.4. The method of claim 1, wherein the set of 2D medical images isautomatically pre-processed such that important or critical features ofthe patient specific anatomic feature are made visible within the 3Dprintable model.
 5. The method of claim 4, wherein pre-processing of the2D medical images is based on the specific anatomic feature, specificpathology of the patient or any downstream application such aspre-operative planning or training purpose.
 6. The method of claim 1,wherein the set of 2D medical images is pre-processed to generate a newset of 2D medical images which are evenly distributed according to apredetermined orientation.
 7. The method of claim 6, wherein thepredetermined orientation is based on the patient specific anatomicfeature, specific pathology of the patient or any downstream applicationsuch as pre-operative planning or training purpose.
 8. The method ofclaim 6, wherein the predetermined orientation and spacing between each2D medical image within the new set of 2D medical images are determinedusing machine learning techniques.
 9. The method of claim 6, wherein thepredetermined orientation and spacing between each 2D medical imagewithin the new set of 2D medical images are user configurable.
 10. Themethod of claim 1, wherein a missing slice from the set of 2D medicalimages is automatically detected.
 11. The method of claim 10, wherein a2D image corresponding to the missing slice is generated usinginterpolation techniques.
 12. The method of claim 1, wherein thesegmentation technique is based on one or a combination of the followingtechniques: threshold-based, decision tree, chained decision forest anda neural network method.
 13. The method of claim 1, wherein voxel basedclassification technique is used in which voxel information from eachaxis or plane is taken into account.
 14. The method of claim 1, whereina likelihood of a voxel of the 3D image having properties similar to thepatient specific anatomic feature is calculated using a logistic orprobabilistic function.
 15. The method of claim 1, wherein a neuralnetwork determines a weight for each axis or plane in each voxel of the3D image.
 16. The method of claim 1, wherein the segmentation techniqueis further improved using multi-channel training. 17-18. (canceled) 19.The method of en claim 1, wherein a 3D mesh model of the patientspecific anatomic feature is generated from the segmented 3D image, andthe 3D printable model is generated from the 3D mesh model. 20.(canceled)
 21. The method of claim 1, wherein points or areas in the 3Dmesh model requiring further post processing steps are automaticallydetected.
 22. The method of claim 1, wherein further post processingsteps include placement of a dowel or other joining structure, andwherein the placement of the dowel or other joining structure isautomatically determined.
 23. (canceled)
 24. The method of claim 1,wherein parameters of the patient anatomic feature are automaticallydetermined from the analysis of the generated 3D image, such as volumeor dimensions of the anatomic feature, thicknesses of the differentlayers of the anatomic feature.
 25. The method of claim 1, wherein thespecific anatomic feature is a heart and the measured parameters includeone of the following: volume of the heart, volume of blood in eachchamber of the heart, thickness of the different layers of the heartwall, size of a specific vessel. 26-27. (canceled)
 28. The method ofclaim 1, wherein a 3D mesh is generated from the set of 2D medicalimages, in which the 3D mesh is a polygonal representation of the volumeof the patient specific anatomic feature.
 29. The method of claim 28,wherein a line or spline is extracted from the 3D mesh along a directionof the patient specific anatomic feature.
 30. The method of claim 28,wherein a classifier is used to identify the anatomic feature from theextracted line or spline, identify the physical properties of theanatomic feature from the extracted line or spline, and/or identify apathology of the anatomic feature from the extracted line or spline.31-33. (canceled)
 34. The method of claim 30, wherein the classifier istrained to identify a specific anatomical feature, trained to determineparameters of the specific anatomic feature such as its locationrelative to the human body, dimension or thickness, and/or trained todetermine a potential defect or pathology of the specific anatomicfeature. 35-46. (canceled)
 47. A computer implemented system forgenerating a 3D printable model of a patient specific anatomic featurefrom a set of 2D medical images, the system comprising a processorconfigured to: automatically generate a 3D image from the set of 2Dmedical images; segment the generated 3D image using a machine learningbased image segmentation technique; and output a 3D printable model ofthe patient specific anatomic feature from the segmented 3D image.48-61. (canceled)